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Mathematical learning environments give domain-specific and immediate feedback to students solv- 
ing a mathematical exercise. Based on a language for specifying strategies, we have developed a 
feedback framework that automatically calculates semantically rich feedback. We offer this feed- 
back functionality to mathematical learning environments via a set of web services. Feedback is 
only effective when it is precise and to the point. The tests we have performed give some confidence 
about the correctness of our feedback services. To increase confidence in our services, we explicitly 
specify the properties our feedback services should satisfy, and, if possible, prove them correct. For 
this, we give a formal description of the concepts used in our feedback framework services. The 
formalisation allows us to reason about these concepts, and to state a number of desired properties 
of the concepts. Our feedback services use exercise descriptions for their instances on domains such 
as logic, algebra, and linear algebra. We formulate requirements these domain descriptions should 
satisfy for the feedback services to react as expected. 

1 Introduction 

We use strategies to calculate semantically rich feedback for students solving a mathematical exercise (9]|. 
For example, we can calculate a hint, or show a complete derivation for a number of mathematical do- 
mains, such as prepositional logic, linear algebra, and arithmetic. A strategy captures expert knowledge 
about how to solve a particular problem. It describes which steps a student can take to solve an exercise, 
and in what order. When a student solves an exercise stepwise, we can check whether or not a step 
follows the strategy. 

We have developed an embedded domain-specific strategy language in which we can specify strate- 
gies, and designed a feedback framework on top of it. This framework is used to give detailed feedback 
to interactive mathematical environments such as ActiveMath Ifl4l . the Freudenthal Digital Mathematics 
Environment Q, and our own tool for rewriting logic expressions [13]. A set of feedback services Q 
gives mathematical environments access to our feedback functionality. 

Feedback tops the list of factors leading to good learning [2 ], but it is only effective when it is precise 
and to the point. Feedback messages should not mislead a student practicing an exercise. Therefore, 
we want to ensure that the given feedback is to the point and relevant. The software we have developed 
is augmented with numerous (unit) tests to test correct behaviour. However, a formal definition of the 
concepts used, such as the strategy language, exercises, and the services, contributes to the goal of 
delivering proper feedback. In addition, we give a formulation, and if possible a proof, of the properties 
they satisfy. The formalisation makes the concepts precise, which enables us to reason about them. The 
feedback services use exercise descriptions for their instances on domains such as logic, algebra, and 
linear algebra. The exercise descriptions contain the strategy for solving a particular exercise, a predicate 
that determines whether or not an exercise is solved, etc. The desired properties of our feedback services 



H. Kirchner, C. Munoz (Eds.): First International Workshop on 
Strategies in Rewriting, Proving, and Programming (IWS 2010) 
EPTCS 44, 2010, pp. 21-TJ4] doi ll0.4204/EPTCS.44.2l 



© A. Gerdes, B. Heeren & J. Jeuring 



22 



Properties of Exercise Strategies 



lead to requirements for the exercise descriptions. We will formulate these requirements, and show how 
our formalisation helps in verifying that the requirements are satisfied for a particular exercise. 
This paper makes the following contributions: 

• We give a formal definition of the concepts we use in our feedback framework. 

• We formulate the properties we want our feedback services to satisfy. 

• We formulate the requirements that an exercise description should fulfil, and we show these re- 
quirements are satisfied by an example exercise description. 

This paper is organised as follows. Section|2]introduces the fundamental concepts of our framework, 
such as rewrite rules and strategies. Section [3] lists the services we offer, and formulates the properties 
that we want our services to satisfy. Section [4] states a number of exercise-specific properties that an 
exercise description should have. Section [5] gives related work, and Section[6]concludes. 

2 Strategy Language 

This section gives a formal definition of our rewrite-strategy language. We start the introduction of our 
rewrite-strategy language for specifying exercises with an example. Consider the problem of rewriting 
the expression (a 3 • a 4 ) 2 as a power of a, using the following three rewrite rules: 



A possible strategy to solve the given power expression is to apply these rules bottom-up, until none of 
the rules can be applied anymore. Applying this strategy to the given expression results in the following 



to solve an exercise involving powers. In addition to a strategy, an exercise description consists amongst 
others of the rules that can be applied to a term, the form of the term that is shown to the user (the exercise 
from the viewpoint of the user), a predicate for determining whether or not an exercise is solved. 

In the remainder of this section we give a formal definition of our strategy language and related 
concepts. 

2.1 Strategy Language Definition 

Definition 2.1. Let p be the fixed-point combinator p f = f (p f). A strategy is an element of the 
language of the following grammar: 

a ::= a \ a <£> a \ a <f> a | y | 8 \ I a \ p.f 
a ::= r | ~a 

The components of the grammar for o are called strategy combinators [8]. Two (sub)strategies can be 
combined into a strategy using the sequence ( <£> ) or choice ( <|> ) combinator, with y (always succeeds) 
and 8 (always fails) as unit elements, respectively. A strategy can be tagged with a label (I). The p 
combinator returns the fixed-point of a function that takes a strategy as argument and returns a strategy. 
The non-terminal symbol a is either a rewrite rule r or an applicability check ~<7 parametrised over a 
strategy. 



a x - a y =a x+ y 

(a x ) y =a xy 
(a-b) x = a x -b x 



[AddExp] 
[MulExp] 
[DistExp] 



derivation: (a 3 - a 4 ) 




a . A strategy captures expert knowledge, in this case how 
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This definition corresponds to the definition of a context-free grammar (CFG), extended with fixed- 
points, with an alphabet consisting of rewrite rules and applicability checks. We distinguish two kinds 
of rules: minor rules, also called administrative rules, and major (normal) rewrite rules. Minor rules 
are used to perform administrative tasks, such as moving down into a term, updating an environment, or 
automatically simplifying a term, such as replacing x + by x. A major rule may be turned into a minor 
rule to decrease the granularity of intermediate steps (e.g., rewriting 3 + 4 to 7 in our running example), 
or increase the difficulty of an exercise. The function isMinor is used to determine whether or not a 
rewrite rule is minor. When tracking a student working on an exercise we maintain an environment, for 
example for storing extra information. Major rules typically are the rules a user applies, such as the three 
rules for manipulating powers given above. The example derivation given above only shows the major 
rules. The minor rules which move the focus into a term, for example for moving from (a 3 ■ a 4 ) 2 to a 3 ■ a 4 
to apply rule AddExp, are not shown in the derivation. The author determines whether or not a rule is 
major or minor. It is advisable to make only rules which the user can apply major, since major rules will 
be shown to the user in derivations, and be given as hints. For example, if the focus in the editor cannot 
be set by the user, it is unwise to make a rule that changes the focus in a term a major rule. 

The main purpose of our feedback framework is to track student behaviour, and to automatically 
calculate feedback based on the strategy and the current term. For this purpose we have to find out where 
in a strategy an error is made, for example. Error messages or hints depend on the position in the strategy 
at which a student has arrived. To connect an error message to a particular position in a strategy we label 
a position in the strategy. A labeled strategy can be transformed to a non-labeled strategy with additional 
minor rules, namely Enter and Leave. The Enter and Leave minor rules update the environment in 
order to keep track where we are in a strategy. 

The language of a strategy is the set of sequences of rewrite rules and applicability checks returned 
by function L: 



For example, using Haskell notation for representing lists and omitting minor rules, the list [ AddExp, MulExp ] 
is an element of the language of the strategy for solving power exercises. We will give a more detailed 
example in Section |2"31 

2.2 Derived Combinators 

On top of the primitive strategy combinators given in the previous subsection, we define a set of de- 
rived strategy combinators. These derived strategy combinators are very useful for formulating rewrite 
strategies for exercises. They are be built on top of the basic concepts. 

An important combinator is the left-biased choice (>), which can be defined as follows: 

Oi>a 2 = Oi <|> (~Oi <£> 02) 

The strategy 02 is only considered when G\ is not applicable. Other combinators, such as try, repeat, 
and option, are similar to the well-known EBNF (extended Backus Naur form) constructs. 



L (a) 

L (C7i <£> o 2 ) 
L (oi <|> (T 2 ) 



{a} 

{xy I x £ L(oi),y G L (<T 2 )} 
L(c7i)UL(C2) 

{e} 



Mr) 

L(S) 
L(£o) 

Mm/) 





L(a) 
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option = <|> 7 

try o = O > 7 

repeat a = p (Xx . try (a <3> x)) 

The option strategy combinator applies a strategy optionally. The try combinator applies a strategy when 
it is applicable. The repeat combinator tries to apply a strategy as many times as possible. 

2.3 Navigation 

Besides the derived combinators from the previous subsection, we add a set of traversal combinators to 
our strategy language. Traversal combinators traverse a term, and for example perform rewrite rules or 
strategies somewhere or bottomUp. We use a number of administrative rules for navigating through the 
abstract syntax tree (AST) of an expression: Up, Down, Left, and Right. The minor rule Down takes 
a function as argument, which decides which child to select based on the environment. Using DOWN 
we construct a minor rule that selects all children using the choice combinator: DOWNS. Navigation 
is implemented by means of a zipper ifTOll . which is an efficient data structure to define and move a 
focus in an expression. The zipper can be seen as a combination of an expression and its context. An 
alternative way to navigate is to use position information of (sub)expressions. An implementation using 
this approach uses a list of integers denoting a path from the top of the expression to the subexpression 
in focus. This approach is not as efficient and type-safe as a zipper, since the AST needs to be traversed 
to retrieve the subexpression in focus, and since it is possible to specify paths that do not correspond to 
a position in the tree. 

Many traversal combinators use the once combinator: 

once a = Downs <g> a <g> Up 

The once combinator takes a strategy as argument, and applies it to a direct child of the expression cur- 
rently in focus. After applying once the focus is again at the top-level expression. The once combinator 
returns all possible ways, by means of the choice combinator introduced by DOWNS, in which a strategy 
can be applied once to a direct child of an expression. So an application of a strategy constructed with the 
once combinator may have more than one result, depending on whether or not strategy a is applicable to 
the children. 

The traversal combinator somewhere applies a strategy to a single subexpression (including the ex- 
pression itself). 

somewhere O = p (Xx . a <|> once x) 

If we want to be more specific about where to apply a strategy, we can instead use bottomUp or topDown: 

bottomUp O = p. (Xx . once x\>o) 
topDown o = p (Xx . o t> once x) 

These combinators search for a suitable location to apply an argument strategy in a bottom-up or top- 
down fashion, without imposing an order in which the children are visited. These combinators do not 
apply their argument strategy exhaustively, instead, the argument strategy is applied only once. 

Navigation operators navigate through the abstract syntax of the domain on which the rewrite rules 
are specified. The example rules we presented at the beginning of this section work on algebraic expres- 
sions containing powers. We give a formal definition of the power domain, and the zipper we use for 
navigation on this domain. 
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e ::= v | e \ e e 

(p ::= \e\ \ (p n \ (p-e \ e-(p 

Here v stands for a variable and n for an integer number. In the grammar for the zipper, the expression 
between double square brackets is the expression in focus. The focus can appear inside a power (p n , or 
move left or right into a product ((p-e, ore - (p). The minor rules for navigation are defined by analysing 
the various forms of the zipper. For example, the Up rule is defined as follows: 

Up (p = case (p of [e] ' -> {e'j 

{ei\-e 2 -> {e\ -e 2 j 

e\ ■ {e 2 j -> lei-e 2 j 
^ _^ (up q,)" 

(p-e UP (p-e 

e ■ (p — > e - UP (p 

We return to the example we gave at the beginning of this section. Now that we have a formal 
definition of strategies, we can express the informal strategy in terms of the derived strategy combinators. 

writeAsPowerOf = I {repeat (bottomUp (AddExp <|> MulExp <|> DistExp))) 

Applying the writeAsPowerOf strategy to the expression (a 3 - a 4 ) 2 gives the following derivation: 

[(aW) 2 ] Ka'-a 4 ) 2 } ^ ^-a 4 f APP =^ ECK [a 3 - a 4 ] 2 A ^ XP 

yf ^ [(a 7 ) 2 ] APP =^> ECK I(a 7 ) 2 ] M ^S- XP [a 14 ] APP =^ ECK fa 14 ] [a 14 ] 

The AppCheck (applicability check) is introduced by the repeat and bottomUp combinator. The navig- 
ation rules do not work directly on an expression, but on the zipper containing the expression As a 
consequence, if a strategy uses a traversal combinator it is only applicable to an expression in a context. 
(In the example derivation above we omit the context from the rewrite steps, and only show the expres- 
sion and the focus.) To keep the definition of rewrite rules simple and clean, we have a function that lifts 
a rewrite rule to operate on an expression in a context. The definition of this function is omitted. 



2.4 Semantics 

At the start of this section we defined the language of a strategy. The language contains lists of rewrite 
rules and applicability conditions. In this subsection we define how a strategy transforms an expression. 
We use the following terminology in the definition of the semantics: 

Expression. We only consider expressions generated by a grammar. 

Environment. An environment stores additional information at intermediate steps, such as auxiliary res- 
ults. An environment is a set of key/value pairs, which can be added, removed, consulted, and 
updated. 

Rewrite rule. A rewrite rule r is a binary relation on the product of an environment and an expression: 
(ri xei)-^ (1^2 x e 2 ). A rewrite rule is tagged with a boolean indicating whether or not it is a 
minor rule. If the rewrite rule does not change the environment we use the notation e\ e 2 . 

A lifted rewrite rule is a binary relation on the product of an environment and a zipper (an expres- 
sion in focus together with its context): (ri x <pi) -w x (p2). 
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State. The state of an exercise is modelled as the product of an environment, a zipper, and a strategy. 

To check if we have reached an end state we want to determine whether or not the empty sentence 
is a member of the language of a strategy. For this purpose we define a function empty, similar to the 
function empty defined for CFGs. We first define an auxiliary function that returns all sentences that 
consist of minor rules only: 

Definition 2.2. Function minorSentences returns all sentences in the language of a strategy that consist 
of minor rules only, including the empty sentence e: 

minorSentences (a) = {r\ ... r n \r\ ... r n € L (c), V i € 1 ... n . isMinor (r ( -) } 

We need this function to determine if there are any trailing minor rules after the last major rewrite rule. 
We define the empty function in terms of the minorSentences function: 

Definition 2.3. Function empty checks whether or not the language of a strategy contains the empty 
sentence e, or a sentence consisting of minor rules only: 

empty (a) = minorSentences (a) / 

Both functions are easily lifted to take a state as argument. 

The smallest action that can be performed with a strategy is a step: the application of a rewrite rule. 
Before we define a step relation that performs a state transition, we define a relation that splits a strategy 
into its first rule or applicability check and the remaining strategy. 

Definition 2.4. The relation h-» splits a strategy into a rule or an applicability check and the remaining 
strategy: ffi i->a <g> a 2 . 

Oj i-» a <R> 03 S e L (oj) 02 a <f> 03 

a^a<f>y 

CJi <£> o 2 ^a <£> (03 <£> 02) <5\ <£> 2 ^a <£> 03 

Oi*-^a<f> 03 2 ^a<R>03 f (tl f) i-)- a <£> a 

0\ <|> 2 -> a <g> 03 Oi <|> 2 — > a <g> 03 \lf ^ a <£> a 

f(ji-> Enter I <s> (a <g> Leave i) 

Definition 2.5. The step operator A denotes the relation between the current state S\ and a new state 
S 2 (obtained by applying the rewrite rule r). 

0\ i-> ~a 2 <£> O3 run (Ti x <p! x a 2 ) = 
ai h-> r <£> o 2 (Ti x <pi) (r 2 x <p 2 ) (H x «p! x <7 3 ) A (r 2 x <p 2 x a 4 ) 

(ri x <pi x ai) A (r 2 x <p 2 x a 2 ) (ri x «pi x ai) A (r 2 x «p 2 x a 4 ) 

The run function used in the last rule applies a strategy to a term in a context; its definition is given 
below. 

The step relation — s> ignores whether or not a rule is minor, and it deals with minor and major rewrite 
rules in the same way. Many of the feedback services we have defined ignore minor rewrite rules, since 
we do not want to show such administrative steps to a user. We define a relation similar to step, which 
ignores minor rewrite rules. 
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r 

Definition 2.6. The big step operator -» denotes the relation between a state S\ and a new state S2. The 
new state is obtained by possibly applying minor rules, followed by the application of a single major 
rewrite rule r.Ifr is the last major rule then trailing minor rules, if any, are applied as well. 

S\ A S2 isMinor {r\ ) S2 -» S3 
Si -» S3 

Si A S2 -i isMinor (r) minorSentences (S2) =0 

r 

S\ -» S2 

Si A- S2 -i isMinor (r) m € minorSentences (S2) S2 A S3 

r 

Si -» s 3 

where in is a sequence of minor rules and is the sequential application thereof. 

It is important, when performing a big step, that trailing minor rules are applied. The application of 
trailing minor rules ensure that exhaustively applying the step or big step operator on a term, will end up 
in the same end state(s). This enables us to keep the generated feedback consistent. 

r 

Definition 2.7. The run function is the closure of-», and generates all possible end states from a begin 
state So- An end state contains the empty strategy: 7. 

run (S ) = { (r x (p x 7) | S (r x <p x 7) } 

where -» is the transitive closure of -». 

2.5 Non-determinism 

Almost all exercises can be solved in several, correct ways. For example, consider the expression 
(a 3 ■ a 4 ) 2 again, and suppose it should be solved with the strategy write AsPowerOf which is obtained by 
replacing bottomllp by somewhere in the strategy writeAsPowerOf. 

write AsPowerOf = I {repeat {somewhere (AddExp <|> MulExp <|> DistExp))) 

One of the questions we want to be able to answer is: what is the next step in order to solve the exercise? 
This type of feedback is given by the onefirst feedback service, which we will describe in more detail in 
the next section. In the above example there are two possibilities: DistExp can be applied to the entire 
expression, and AddExp is applicable to the subexpression a 3 - a 4 . Both steps are correct, but which 
do we choose? Making a random choice would make our feedback framework non-deterministic. This 
problem may show up whenever we use the choice combinator, which may also be introduced by other 
combinators (such as DOWNS), to combine various solution strategies. Applying a strategy that uses the 
choice combinator to an expression may result in multiple answers. 

To prevent non-deterministic behaviour we introduce a rule ordering: a total order, denoted by 
<, on rules. For now, the ordering is only on the rules themselves, but one can imagine taking the 
environment into account in the rule ordering. An example rule ordering for the power domain is: 
AddExp < MulExp < DistExp. In this case, the first step in our example is AddExp. 
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2.6 Restrictions 

To use strategies to track student behaviour and give feedback, we impose some restrictions on the form 
of strategies. These restrictions are similar to some of the restrictions imposed on context-free grammars 
to be able to use them for parsing. 

Left-recursion. A context-free grammar is left-recursive if it contains a nonterminal that can be rewrit- 
ten in one or more steps using the productions of the grammar to a sequence of symbols that starts with 
the same nonterminal. The same definition applies to strategies. For example, the following strategy is 
left-recursive: 

leftRecur = /J, (Xx . x <£> AddExp) 

The left-recursion is obvious in this strategy, since x is in the leftmost position in the body of the abstrac- 
tion. Left-recursion is not always this easy to spot. Strategies with leading minor rules may or may not be 
left-recursive. Strictly speaking, these strategies are not left-recursive because the strategy grammar does 
not differentiate between minor and major rules. However, our semantics make it that these strategies 
sometimes display left-recursive behaviour. For example: 

leftRecur' = 11 (Xx . Down <se> x <f> AddExp) 

Here, applying the minor rule DOWN repeatedly will eventually cause the strategy to reach the leaf of 
an expression tree, and stop. Hence this strategy is not left-recursive. However, this is a property of 
DOWN that is not shared by all other minor rules. If we use a minor rule that increases a counter in the 
environment, an action that will always succeed, the strategy is left-recursive. 

Top-down recursive parsing using a left-recursive context-free grammar is difficult. We have chosen 
for top-down recursive parsing because we need to parse a derivation incrementally. Other parsing 
schemes, such as bottom-up parsing, also have their difficulties. A grammar represented in parser com- 
binators ifTTTl is not allowed to be left-recursive. Similarly, for a strategy to be used in our framework, it 
should not be left-recursive. Trying to determine the next possible symbol(s) of a left-recursive strategy 
by means of the split operator will get stuck in a loop. 

Left-recursive strategies are not the only source of non-terminating strategy calculations. The fact 
that our strategy language has a fixed-point combinator implies that we are vulnerable to non-termination. 
The implementation of our strategy language has been augmented with 'time-outs' that will stop the 
execution and report an error message. 

Left-factorisation. Left-factoring is a grammar transformation that is useful when two productions for 
the same nonterminal start with the same sequence of terminal and/or nonterminal symbols. This trans- 
formation factors out the common part of such productions. In a strategy, the equivalent transformation 
factors out common sequences of rewrite rules from sub-strategies separated by the choice combinator. 

A strategy that contains left-factors may be problematic. Consider the following, somewhat con- 
trived, strategy: 

leftStrat = i\ (AddExp <£> MulExp) <|> l 2 (AddExp <£> DistExp) 

The two sub-strategies labeled with l\ and l 2 have a left-factor: the rewrite rule AddExp. After the 
application of AddExp, we have to decide which sub-strategy to follow. Either we follow sub-strategy 
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l\, or we follow sub-strategy l 2 . Committing to a choice after seeing an application AddExp is unfor- 
tunate, since it will force the student to follow the same sub-strategy. For example, if i\ is chosen after 
an application of AddExp has been submitted, and a student subsequently takes the DistExp step, we 
erroneously report that the step does not follow the strategy. Therefore, left-factorising a strategy is de- 
sirable, since we do not want to commit early to a particular sub-strategy. The example strategy can be 
left-factored as follows: 

leftStrat' = AddExp <&> (MulExp <|> DistExp) 

It is clear how to left-factor (major) rewrite rules, but how should we deal with labels, or minor rules in 
general? In the remainder of this section we focus on how to deal with labels. Pushing the labels inside 
the choice combinator, 

leftStrat 1 = AddExp <£> (l\ MulExp <j> £ 2 DistExp) 

or making a choice between the two labels probably breaks the relation between the label and the strategy. 
Recall that labels are used to mark positions in a strategy, which is connected to a feedback text. 

A different way to deal with left-factors uses backtracking. With backtracking we remember the start 
position of the left-factor in the strategy. In the case that the chosen sub-strategy fails we roll back and 
continue with a different sub-strategy. However, using backtracking it is possible to guide a student to a 
dead end (a sub-strategy that fails). This violates our goals of giving proper and relevant feedback. 

Our solution is to detect and report left-factors. The detection of left-factors in a strategy is relatively 
straightforward. The detection and notification enables strategy developers to construct strategies without 
left-factors. 

3 Services 

Our feedback services offer a wide variety of feedback functionality, so that learning environments using 
our services can provide different kinds of feedback. A learning environment can amongst others ask 
for the following kinds of feedback: Is the student still on the right path towards a solution? Does the 
step made by the student follow the strategy for the exercise? What is the next step the student should 
take? Has the student made a common error? What does a worked-out solution look like? This section 
formalises the corresponding services. 

3.1 Exercises 

Most of the feedback services calculate feedback based on a strategy, but sometimes there are other com- 
ponents that play a role in our services. All components necessary for our feedback services are grouped 
together in an exercise. An exercise contains all domain-specific (and exercise-specific) functionality, 
together with an exercise code for identification. The most important component of an exercise is its 
strategy. Additional rewrite rules can be added to an exercise to help detect possible detours. We not 
only specify proper rewrite rules, but also buggy rules. A buggy rule captures a common misconception. 
If we detect an application of a buggy rule, we report this to the user. We also need predicates for check- 
ing whether or not an expression is a suitable starting expression that can be solved by the strategy, and 
whether or not an expression is in solved form. For diagnosing intermediate answers, we need an equi- 
valence relation to compare a submission with a preceding expression, and a similarity relation, which 



30 



Properties of Exercise Strategies 



is possibly more liberal than syntactic equality. Although not of primary importance, it can be conveni- 
ent to have a randomised expression generator for the exercise. The last component of an exercise is a 
function that returns the order of rules. 

Definition 3.1. An exercise consists of an identification code, a strategy o, a rule set {r\ ... r n }, a buggy 
rule set, an equivalence relation =, a similarity relation a predicate isSuitable, a predicate isReady, 
an optional expression generator, and a rule ordering function <. 

An example of an exercise is the exercise for our running example: calculating with powers. 

powerExercise = (powerExercise 

,£ (repeat (bottomUp (AddExp <|> MulExp <|> DistExp))) 

,K = ;pj [ReciExp]} 

,{a x -a y = a xy [BUGADDEXP]} 

, eqPower, simPower, suitablePower , readyPower 

, DistExp < MulExp < AddExp) 

Here, eqPower, simPower, suitablePower, and readyPower are defined as follows, where ~ stands for 
syntactic equality: 

normPower v = v 

normPower (e\ ■ e-i) = simplifyPower (normPower e\ ■ normPower ej) 
normPower (e n ) = simplifyPower ((normPower e) n ) 

simplifyPower ((a x ) y ) = a xy 
simplifyPower (a x ■a y ) = a x+y 
simplifyPower ((a ■ b) x ) = a x -b" 
simplifyPower e = e 

eqPower e\ e^ = normPower e\ — normPower e2 
simPower e\ e 2 = e\ ~ e 2 
suitablePower e = normPower e e 
readyPower e = normPower e~e 

We have simplified the implementation of normPower by ignoring the fact that • is associative. In this 
example, we have chosen to use normalisation as a base to define the exercise-specific functions, such as 
=. However, this is not necessarily the case for other domains. 

3.2 Formalised Services 

We define the set of services we offer to learning environments in terms of the relations defined in the 
previous section. 

allfirsts. The allfirsts service returns all next steps that are allowed by a strategy in a particular state: 

r 

allfirsts So = { (S, r) \ So -» S} 

Consider the following state S: 

S = (r, l(a 3 ■ a 4 ) 2 }, ((somewhere AddExp) <e> MulExp) <|> 

(DistExp <g> repeat MulExp <g> AddExp)) 
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An allfirsts S service call gives the following result. The Up minor rule is introduced by the 
somewhere combinator to get the focus back to its original place. 

{((r,([a 7 l) 2 , Up <£> MulExp),AddExp) 

, ((r,[(a 3 ) 2 • (a 4 ) 2 ], repeat MulExp <3£> AddExp),DistExp) } 

onefirst. The onefirst service returns a single possible next step that follows the strategy. This service 
uses the allfirsts service and the rule ordering. 

onefirst So = (S,r) where (S,r) G allfirsts So A V (S,,r,') G allfirsts So ■ r ^ r,- 

Performing a onefirst service call on the state S from the previous paragraph, taking into account 
the rule ordering from subsection 12.51 gives following result: 

((T,([[a 7 ]]) 2 ,Up <e> MulExp), AddExp) 

derivation. The derivation service returns a worked-out solution of an exercise starting with the current 
expression. 

derivation Sq = (S\,ri) (S2> r 2) • • • (S n ,r n ) where empty (S n ) A 
V i G 1 ... n . [Si-i , r,) = onefirst Sj 

ready. The ready service checks if the expression in a state is in a form accepted as a final answer. The 
ready service is an interface to the isReady predicate defined in an exercise. 

stepsremaining. The stepsremaining service computes how many steps remain to be done according to 
the strategy. This is achieved by calculating the length of a derivation. 

apply. The apply service applies a rule to an expression at a particular location, regardless of the strategy. 
The location is represented as a list of integers, where each integer n represents the number of steps 
to the right after a step downwards in a subexpression (the nth child). Starting at the root of an 
expression we can assign every subexpression a unique location. 

The function setFocus translates a location to a sequence of minor rules that puts the focus at a 
particular subexpression. 

setFocus So [ ] = So 

setFocus So (n : ns) = setFocus S\ ns where So ' -> Si 

The function focusToRoot sets the focus to the root of an expression. We omit the definition of this 
function. The apply service is defined as follows: 

apply r loc So = S\ where setFocus loc (focusToRoot So) A Si 

For example, apply MulExp [1] (T, [(a 3 ) 2 • (a 4 ) 2 ], a) gives (r,((a 3 ) 2 - [a 8 ]), a). 

applicable. The applicable service takes an expression and a location in this expression, and returns all 
major rules that can be applied at this location, independent of the strategy. Let R be the union of 
the rules in the strategy and the exercise rule set, then applicable is defined as follows: 

applicable loc So = {r \ r G R,Si A S2} where Si = setFocus loc (focusToRoot Sq) 
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generate. The generate service takes an exercise code and a difficulty level (optional), and returns an 
initial state with a freshly generated expression. 

diagnose. The diagnose service diagnoses an expression submitted by a student. Possible diagnoses are: 

• NotEq: the current and submitted expression are not equivalent, i.e. something is wrong, 

• Buggy: a common misconception has been detected, 

• Similar: the expression is similar to the last expression in the derivation, 

• Expected: the submitted expression is expected by the strategy, 

• Detour: the submitted expression was not expected by the strategy, but the applied rule was 
detected, 

• Correct: the submitted expression is correct, but we cannot determine which rule was ap- 
plied. 

The diagnose service is defined as follows, where the unFocus function converts a focussed ex- 
pression to a normal expression: 

diagnose (T, <p, a) new = 

if unFocus <p ^ new then 

b 

if 3 b G buggyRuleSet .3e. unFocus <p e A e = new then Buggy 

else NotEq else 

if unFocus (p ~ new then Similar else 

ii new G allfirsts (T,(p,o) then Expected else 

if 3 r G ruleSet . 3 e . unFocus (p e A e = new then Detour 

else Correct 

We conclude this section with a soundness result. We give a theorem connecting the language 
concept, the function L, to the services built on top of strategies defined in this section. We start with the 
introduction of a lemma that simplifies the proof of the theorem. 

Lemma 3.2. A major language Jz? is the language of a strategy without occurrences of minor rules: 

Jz? (a) = { [a | a ^— as, -i isMinor a\\as G L (a) } 

Let F be the set of all splits given by the split relation for o: { (7, 1 (7 i— > a, }. Then the major language of 
a without the empty sentence is equal to the union of major languages of the elements in F: 

J? {a) \{e}^ U Sf(a f ) 

o f eF 

Proof. By case analysis on the structure of a. □ 

Theorem 3.3. Let So be a state (T x <p x a) andr be a sequence of rules [r\ ... r n ] such that derivation (So) 
(S u n) ... (S n ,r n ). Then r G JSf (a). 

Proof. We sketch a proof. Via the definitions of derivation, onefirst, and allfirsts we know that for every 
element in the derivation sequence [r\ ... r n ]& big step has to be performed. We distinguish two cases 
in our hypothesis [r\ ... r n ] G J*f (a): 

?i = : From the statement above we deduce that the strategy a is either y or consists of a strategy with 
minor rules only. From the definition of the major language Jz? we know e G Jz? (a). 

n > : This case is proven by induction on the length (n) of the sequence [r\ ... r n ] using Lemma [3721 

□ 
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4 Exercise Properties 

The following lemmas express properties that the components in an exercise should have. The lemmas 
connect the various components of an exercise. The proof of these lemmas for a particular exercise 
indicates the soundness of the corresponding exercise specification. 

Lemma 4.1. Let So be a state (r, [e] , cr) and derivation So = {S\,r\) ... (S n ,r n ). If e is a suitable start 
expression, the expression in the last state (S n ) is ready. 

Lemma 4.2. Let r be a rewrite rule for a particular exercise: e\ ~» e%. Then r is semantics preserving, 
i.e., e\ = ^2- 

Corollary 4.3. Let So be a state and derivation So = (Si,r\) ... (S n ,r n ). Then all expressions in So up 
to S n are in the same equivalence class determined by the exercise 's equivalence relation =. 

Lemma 4.4. Let E be the set of expressions generated by the exercise's generator. All expressions e G E 
are suitable and not ready: suitable (e) A -i ready (e). 

Lemma 4.5. Let So be a state, onefirst So = and allfirsts So = {(S2,rz) ■■■ (S n ,r n )}. Then 

r\ = min {r-i ... r n } with respect to the rule ordering. 

The proofs of these lemmas for the exercise for calculating with powers, are rather easy. This is 
because the rules that are used in the strategy are the same as the rules used to calculate the normal form 
of a power expression. We provide the proofs for these lemmas in an accompanying technical report (H. 

5 Related Work 

The strategy language on which our work is based is very similar to languages that are used in parser 
libraries lfl5l . Some differences, such as the usage of labels, the focus on intermediate answers, and the 
concept of minor rules, make the existing libraries unsuitable for generating feedback. 

Our language is also similar to strategic programming languages such as Stratego [16] and Elan J4]|, 
and tactic languages used in theorem proving. We compare our approach with two more formal ap- 
proaches to strategies. Kaiser and Lammel construct a mechanised formal model of Stratego [12J. Our 
strategy language is different from Stratego in the sense that we, in addition to the final term, also focus 
on the intermediate rewrite steps. We do not focus on the development of a mechanised model: instead, 
we give a formal definition of our feedback services that use our strategy language. Tacticals, proof plans 
and methods Q are used to automatically prove theorems. On an abstract level, these plans and methods 
play the same role as strategies: we can view a strategy as a proof plan for proving that an answer is the 
solution to an exercise. Aspinall et al. (H introduce the tactic language Hitac that can be used to con- 
struct hierarchical proofs, so called hiproofs. To evaluate Hitac programs two semantics are given: a big 
step semantic that captures the intended meaning and a small step semantic that covers the details of the 
proof. As far as we found, the tactic language is not used to generate feedback, or to recognise proving 
steps made by a student. Moreover, we provide a set of services that enables learning environments to 
access our functionality. 

6 Conclusions 

In this paper we have presented a formal and precise definition of the main concepts that we use to 
construct semantically rich feedback in learning environments. In addition to a precise definition of our 
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strategy language we define a step, and a big step relation. These relations give the semantics of the 
strategy language. Feedback services are an interface to our feedback functionality that can be used by 
learning environments. These services are expressed in terms of the big step relation. 

Our formalisation gives us more confidence in the design choices we have made. Furthermore, 
we can now validate our current implementation using the properties we state. In the future we want 
to extend the work in this paper by providing proofs for other domains, such as linear algebra and 
propositional logic. 
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